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Abstract 

In  this  paper,  the  fusion  of  hyperspectral  imaging  (HSI)  sensor  data  and  high-resolution 
panchromatic  imagery  (HPI)  is  explored.  Spatial/spectral  sharpening  techniques  have 
been  proposed  to  increase  the  spatial  resolution  of  spectral  imagery  with  HPI.  The  intent 
of  the  process  is  to  reconstruct  high  spatial  resolution  imagery  with  high  spectral  fidelity. 
Target  detection  and  identification  of  such  imagery  can  be  achieved  with  spectral  and 
spatial  processing  techniques. 

Three  sharpening  approaches  are  reviewed  here.  The  first  one  uses  pseudo-inverse 
techniques,  the  second  and  third  are  component  replacement  methods  in  spatial  and 
frequency  domains,  respectively.  The  performance  of  these  approaches  are  assessed  and 
shown  with  examples.  In  addition,  a  spatial-spectral  analysis  with  fusion  of  HSI  and  HPI 
is  also  demonstrated  for  enhanced  target  detection  and  identification. 

1  Introduction 

Conventional  single  band  or  multi-spectral  sensors  typically  image  in  wide,  discrete 
spectral  bands  with  fine  spatial  resolutions.  In  contrast,  hyperspectral  imaging  sensors 
collect  image  data  in  hundreds  of  contiguous,  narrow  spectral  bands  (~10  nm)  with 
coarser  resolutions.  The  increased  number  of  sensor  bands  provides  higher  spectral 
resolution  and  more  opportunities  to  detect  subtle  spectral  differences  in  signatures  that 
are  too  narrow  to  be  differentiated  in  multi- spectral  imagery.  For  each  ground  sample 
distance  (GSD)  pixel  within  a  hyperspectral  image,  a  continuous  spectrum  is  measured 
and  used  to  identify  materials  by  their  reflectance  or  emissivity.  On  the  other  hand, 
signal-to-noise  ratios  (SNR)  of  the  HSI  data  tend  to  be  sub-optimal.  To  maintain  certain 
SNR,  HSI  resolutions  are  in  general  coarser  than  broadband  image  resolutions.  If 
hyperspectral  image  can  be  spatially  “sharpened”  via  the  broadband  high-resolution 
image,  the  overall  performance  is  expected  to  improve  further. 


'  This  work  was  sponsored  by  the  Department  of  Defense  under  Air  Force  Contract  F19628-95-C-0002. 
Opinions,  interpretations,  conclusions  and  recommendations  are  those  of  the  authors  and  not  necessarily 
endorsed  by  the  United  States  Air  Force. 
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Currently,  there  are  a  number  of  space-based  hyperspectral  measurement  platforms  to  be 
launched  in  the  foreseeable  future:  NASA’s  EO-1,  Air  Force’s  Warfighter- 1  and  Navy’s 
NEMO  (Naval  Earth  Map  Observer).  Their  characteristics  are  summarized  in  Table  1. 
Note  that  in  addition  to  the  HSI  sensor,  a  high-resolution  panchromatic  imaging  (HPI) 
sensor  is  also  onboard  for  each  of  these  systems,  with  the  linear  spatial  resolution  of  3  to 
8  times  better.  It  is  important  that  effective  approaches  of  hyperspectral  image 
sharpening  with  the  high-resolution  broadband  image  be  explored  to  enhance  the  utility 
of  future  HSI  data. 

A  number  of  approaches  have  been  attempted  and  applied  to  multi-spectral  images.  In 
Section  2,  three  approaches  are  briefly  reviewed  and  implemented:  the  pseudo-inverse 
technique,  color  normalization,  and  spatial  frequency  correction  methods.  Enhanced 
target  detection  and  identification  is  illustrated  in  Section  3  with  additional  spatial- 
spectral  analysis  of  fusion  of  high-resolution  information. 

2  Review  and  Demonstration  of  Sharpening  Approaches 

In  this  section,  three  techniques  for  image  sharpening  are  first  reviewed.  A  sample  data 
is  created  based  on  field  data.  The  algorithms  are  then  applied  to  the  generated  HSI  and 
HPI  data  for  a  sharpened  HSI.  Algorithm  performance  is  then  evaluated  by  comparing 
with  the  original  truth  data. 

2.1  Approaches 

Three  sharpening  approaches  discussed  here  are:  pseudo-inverse  techniques,  and 
component  replacement  methods  in  spatial  and  frequency  domains,  respectively. 

2. 1. 1  Pseudo-inverse  technique  ^ 

Given  a  high-resolution  panchromatic  image  (HPI)  co-registered  with  a  set  of 
hyperspectral  images,  a  system  of  equations  can  be  established  for  the  reconstruction  of  a 
high  spatial  resolution  HSI  (SHSI).  The  value  at  a  pixel  in  HPI  is  the  spectral  sum  at  the 
same  pixel  in  SHSI  and  the  spectral  value  at  a  pixel  in  HSI  is  the  pixel-sum  over  the  HSI 
GSD  in  the  same  spectral  band  of  SHSI.  If  the  sharpening  ratio  is  r  and  the  number  of 
spectral  bands  is  b,  then  the  number  of  equations  over  the  GSD  of  HSI  is  r^  -i-  b,  and  the 
number  of  unknowns  for  the  SHSI  reconstruction  is  r^  x  b.  As  the  sharpening  ratio  and 
the  number  of  spectral  band  increase,  the  number  of  unknowns  increases  faster  than  the 
number  of  additional  equations.  The  system  of  equations  is  under  determined  for  a 
unique  solution.  Pseudo  matrix  inversion  algorithms  are  applied  to  obtain  a  solution  with 
least-mean-squared  (LMS)  estimations  for  the  fusion  problem.  In  this  approach,  it  is 
important  that  the  panchromatic  image  is  in  perfect  registration  with  HSI.  The  pseudo 
inverse  technique  requires  singular  value  decomposition.  In  addition,  the  LMS 
estimations  based  on  the  under  determined  system  of  equations  may  not  achieve  the 
required  spectral  accuracy. 


2. 1.2  Color  Normalization 


The  color  normalization  algorithm  conventionally  used  in  MSI  ^  is  modified  for  HSI 
sharpening.  It  multiplies  each  of  the  HSI  bands  by  the  HPI  and  the  resulting  values  are 
each  normalized  by  the  averaged  HPI  data  over  the  number  of  pixels  equivalent  to  the 
GSD  of  HSI.  It  is  defined  by  the  following  equation: 

SCNi  =  (HSIi  *  HPI)  /  ave(HPI)GSD 

where  HSp  is  the  HSI  band  and  SCNi  is  the  sharpened  image  band  by  color 
normalization. 

The  approach  is  straightforward  in  merging  the  spatial  contrast  of  HPI  into  each  spectral 
band  of  HSI.  Similar  to  the  pseudo-inverse  technique  approach,  it  also  requires  precise 
HSI/HPI  registration  and  the  reconstructed  data  may  not  achieve  great  spectral  accuracy. 

2. 1.3  Spatial  Frequency  Correction 

Some  sharpening  approaches  use  wavelet  transformations"^.  Images  are  decomposed  into 
different  spatial  frequency  scales.  For  each  of  the  HSI  bands,  the  high  frequency 
components  are  replaced  with  components  from  HPI.  The  sharpened  image  is  then 
obtained  via  inverse  transformation  of  the  modified  spectra.  In  practice,  2-D  Fourier 
transformation  can  be  applied  for  image  spatial  frequency  analysis.  The  sharpening 
process  is  described  as  below: 

SFCi  =  FFT  ^  {FFT  (HSIOiow  +  FFT(HPI)hi } 

where  SFCi  is  sharpened  HSh  by  frequency  correction,  FFT  (HSIi)iow  represents  low 
frequency  components  of  HSh  and  FFT(HPI)hi  represents  high  frequency  components  of 
HPI 

In  this  approach,  the  spatial  shift  due  to  imperfect  HSI/HPI  registration  will  show  up  as  a 
phase  shift  in  the  frequency  spectrum.  In  addition,  careful  selection  of  the  number  of 
high  frequency  components  for  replacement  is  required  to  avoid  artifacts  in  the  sharpened 
image.  Once  again,  some  loss  of  spectral  quality  is  expected  in  the  reconstructed  data. 

2.2  Demonstration  Example 

The  review  in  Section  2.1  indicates  that  all  three  approaches  require  good  co-registration 
between  HSI  and  HPI.  Since  HSI  and  HPI  commonly  co-exist  on  the  same  platform,  co¬ 
registration  to  within  a  small  fraction  of  an  HPI  pixel  is  expected.  For  performance 
comparisons,  perfect  co-registration  is  assumed  here.  Future  work  will  address  the  co¬ 
registration  issues.  A  set  of  simultaneous  HSI  and  HPI  data  is  generated  and  applied  with 
the  sharpening  methods.  A  HYDICE  (Hyperspectral  Digital  Image  Collection 
Experiment)  major  frame  is  used  as  the  “truth”  in  both  spatial  (0.8  m/pixel)  and  spectral 
(AA.  ~  10  nm)  domains.  This  input  “truth”  data  is  then  processed  to  generate  the 


contemporaneous  test  data  of  HSI  (8  m/pixel,  AA,  ~  10  nm)  and  HPI  (0.8  m/pixel,  wide 
band).  The  first  major  frame  of  HYDICE  data  from  Forest  Radiance  I  Run  05  is  used. 
The  frame  is  a  portion  of  a  forest  scene.  It  contains  camouflage  nets  (>  lOmx  10  m) 
and  several  vehicles  (  ~  4  m  x  8  m).  The  image  has  320x320  pixels,  0.8  m  per  pixel.  25 
bands  from  0.63  (tm  to  0.9  (tm  are  used.  The  HPI  data  is  generated  with  a  25-band 
integration.  It  is  a  320x320  single-band  image  with  0.8  m  per  pixel.  The  HSI  data  for 
test  is  obtained  by  under-sampling  the  spatially  blurred  input  data.  The  FWHM  (full 
width  half  maximum)  of  the  blurring  point  spread  function  (PSF)  is  5 -pixel  wide  and  the 
spatial  under-sampling  is  5  to  1.  The  resulting  HSI  has  25  bands;  each  band  is  64  x  64 
pixels  in  size  and  4  m  per  pixel.  Figure  1  depicts  the  flow  of  data  generation  for  HSI  and 
HPI  fusion. 

Results  of  sharpening  using  the  three  approaehes  are  shown  in  Figure  2.  All  three  images 
are  visually  sharper  than  the  low  resolution  HSI  and  comparable  to  the  original  input  of 
HSI  (“the  truth”).  Two  measures,  spectral  angle  and  spectral  distance  (Euclidean 
distance),  are  used  to  evaluate  the  sharpening  algorithms.  These  are  calculated  as: 

Spectral  Angle  (A,B)  =  COS  \A  .  B  /  |A|  |B|) 

Spectral  Distance  (A,B)  =  |A  -  B| 

where,  A  and  B  are  two  multi-band  spectra. 

A  zero  angle  and  zero  distance  represent  a  perfect  match  of  the  two  spectra.  The 
sharpened  images  are  compared  to  “the  truth”  data  in  terms  of  speetral  angle  and  speetral 
distance.  The  spectral  distance  normalized  by  the  pixel  amplitude  in  the  truth  image  is 
caleulated  for  eomparison.  The  frame-averaged  differenees  are  listed  in  Table  2.  For 
comparison,  the  difference  measurements  from  the  unsharpened  HSI  are  also  included  in 
the  table.  It  shows  that  the  sharpened  images  improve  signifieantly  from  the  unsharpened 
HSI  in  speetral  distanee,  but  shows  no  obvious  improvement  in  spectral  angle.  Among 
the  sharpened  images,  color  normalization  (SCN)  is  closer  to  the  truth  in  spectral  angle 
than  results  of  pseudo-inverse  (SPI)  and  frequency  correction  (SFC).  On  the  other  hand, 
the  image  of  frequency  correction  is  the  closest  to  the  truth  in  spectral  distance. 

The  sharpened  images  generally  appear  spatially  sharper.  However,  the  impact  of  such 
results  on  target  deteetion  and  identifieation  is  small.  Sinee  the  combined  HSI  and  HPI 
data  is  severely  under-determined,  spectral  features  of  objects  smaller  than  the  original 
HSI  resolution  cannot  be  fully  resolved  in  the  reconstruetion  of  the  sharpened  HSI.  For 
enhanced  target  detection  and  identification,  additional  spatial  and  spectral  analysis  at 
feature  level,  as  diseussed  below,  is  neeessary.  In  this  paper,  sharpened  HSI  image  by 
color  normalization  will  be  used  for  further  analysis. 

3  Spatial-spectral  Analysis 


^  f.  n 

Various  spatial  and  spectral  analysis  approaches  have  been  attempted  ’  ’  .  Here,  a 
combined  spatial- speetral  analysis  is  earried  out  to  demonstrate  HSI  and  HPI  fusion  for 


enhanced  background  characterization  and  target  detection/identification.  An  analysis 
approach  is  depicted  in  Figure  3.  The  HSI  data  is  first  applied  with  background 
classification  and  anomaly  detection.  The  background  class  statistics  calculated  from 
HSI  are  employed  when  the  sharpened  HSI  is  also  processed  with  background 
classification  and  target  detection.  At  the  same  time,  spatial  edge  detection  for  target  and 
background  boundaries  is  applied  to  the  HPI.  These  edges  are  combined  with  results 
from  the  sharpened  HSI.  Targets  and  backgrounds  are  spatially  defined  by  the  edges. 
Subsequently,  their  materials  can  be  identified  with  spectral  characteristics. 

In  order  to  evaluate  the  analysis  performance,  an  expanded  data  set  is  used  here.  Three 
major  frames  of  HYDICE  data  from  Forest  Radiance  I  Run  05  are  used  in  the 
demonstration  example.  The  original  image  of  300x960  pixels  in  size,  0.8  m  per  pixel 
and  210  bands  (the  “truth”  image)  is  used  as  the  input.  The  simulated  low-resolution  HSI 
is  60x192  pixels  in  size,  4  m  per  pixel  and  210  bands.  The  HPI  is  obtained  with  band 
integration  from  0.4  to  0.8  pm,  300x960  pixels  with  0.8  m  per  pixel.  A  sharpened  HSI 
data  cube  is  constructed  from  the  HSI  and  HPI  with  the  color  normalization  method. 

Figure  4  shows  a  reference  image  (RGB)  of  HSI,  as  well  as  background  classification  and 
anomaly  detection  map  derived  from  the  unsharpened  HSI.  The  classification  is 
performed  with  an  unsupervised  feature  extraction  using  principal  component  analysis 
(PC A).  The  Eigen  space  is  first  determined  for  the  scene  from  the  covariance  matrix  of 
the  data  cube.  It  then  divides  the  scene  into  classes  of  maximal  separations  in  the  Eigen 
space.  In  the  5-class  classification  shown  here,  road,  vegetation  (bright  and  dark),  shade 
and  ground  are  delineated  in  the  scene.  The  background  map  and  spectral  statistics  of 
these  classes  from  HSI  are  employed  in  all  subsequent  processing.  Anomalies  distinct 
from  backgrounds  are  also  detected.  These  detections  are  used  to  cue  for  target  detection 
and  identification  in  the  combined  analysis.  Edges  detected  from  HPI  with  a  SOBEL* 
operator  are  shown  in  Figure  5.  Overlay  of  the  edges  with  background  classification 
from  sharpened  HSI  is  also  shown  in  the  figure.  Background  boundaries  in  the  sharpened 
HSI  are  better  defined  than  in  the  unsharpened  HSI.  These  boundaries  are  further 
enhanced  with  the  edges  derived  from  HPI.  The  image  on  the  right  of  Figure  5  depicts 
the  combined  results  of  spectral  matched  filtering  and  spatial  edges  over  regions  with 
anomaly  detections  from  HSI.  Red  and  green  portions  each  represent  the  detection  of 
different  vehicle  paints.  The  detections  are  in  general  bounded  by  the  edges  shown  in 
blue.  However,  two  regions  of  detection  located  at  the  top  of  the  scene  are  not  well 
defined  in  shape  and  appear  to  be  large  in  size.  These  are  inconsistent  with  features  of 
vehicles.  Further  testing  with  spectral  matched  filtering  with  two  types  of  fabric  confirms 
and  identifies  pieces  of  fabric  in  these  regions  as  shown  in  the  leftmost  image  of  Figure  6. 
An  enlarged  view  of  the  vehicle  detections  is  also  shown  in  the  figure.  The  vehicle  size 
and  orientation  can  be  determined  from  the  bounding  edges.  It  is  classified  as  large  (4x8 
m  )  in  size  if  it  is  4  to  7  pixels  wide  and  8  to  1 1  pixels  long  or  as  small  (3x6  m  )  in  size  if 
it  is  3  to  5  pixels  wide  and  6  to  7  pixels  long.  The  colored  bar  next  to  each  vehicle  in  the 
enlarged  image  depicts  its  type  of  paint,  size  and  orientation. 


4  Summary 


In  this  paper,  sharpening  approaches  were  investigated  and  implemented  on  a  combined 
HSI  and  HPI  data  set.  The  sharpened  HSI  retained  the  spectral  signatures  of  extended 
area  in  HSI  and  in  general  appeared  spatially  sharper.  However,  spectral  features  of 
objects  smaller  than  the  original  HSI  resolution  were  not  fully  resolved  through 
sharpening  due  to  the  under-determined  nature  of  the  combined  data  set.  Further  spatial 
and  spectral  analysis  was  then  demonstrated  to  combine  high-resolution  information  from 
HSI  and  HPI  for  enhanced  background  characterization  and  target  detection  and 
identification.  Anomalies  distinct  from  backgrounds  were  also  detected  from  HSI  and 
used  to  cue  for  target  detection  and  identification  in  the  combined  analysis.  Spatial  image 
processing  was  applied  to  HPI  for  edge  analysis.  As  a  result  of  combined  analysis, 
specific  target  material,  size  and  shape  were  determined. 
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Table  1 


Future  Space-based  Hyperspectral  Measurement  Platforms 


Sensor 

EO-l  (NASA) 

Warfighter- 1  (Air  Force) 

NEMO  (Navy) 

HSI  Spectral 

0.4  -  2.5  pm,  220  bands 

0.4  -  2.5  pm,  200  bands 

0.4  -  2.5  pm,  210  bands 

HSI  EFOV 

30  m 

8  m 

30  m 

Scene  Size 

7.5  kmx  100  km 

5  km  X  20  km 

30  km  X  200  km 

Co-incident  HPI 

10  m 

1  m 

5  m 

Launch 

2000 

2001 

--2001 

Table  2  Sharpening  Algorithm  Performance 


Average  spectral  angle  and  distance  over  1^*  major  frame  of  HYDICE  Forest  Radiance  I  Run  05 


Performance 

Metrics 

Unsharpened 

HSI 

Sharpened 

Pseudo  Inverse 

Frequency 

Correction 

Color 

Normalization 

Spectral  Angle 
COS-*(X.Mi/IXI  IMil) 

4.9° 

8.3° 

5.8° 

bo 

o 

Spectral  Distance"^ 

(X  -  Mi)^  (X  -  Mi) 

21.6% 

13.8% 

8.2% 

10.0% 

*  Spectral  Distance  normalized  to  the  pixel  amplitude  in  the  “truth”  image 


GSD:  0.8  m 
0.4  <  A,  <  2.5  gm 

AX,:10nm  _ 

Truth 


GSD:  4  m 
0.4  <  A,  <  2.5 
Ak :  10  nm 
200  bands 


GSD:  0.8  m 
Single  broad 
band 


HSI 


Figure  1  Flow  diagram  of  data  generation  for  HSI  and  HPI  fusion 


Pseudo  Inverse  Color  Normalization  Frequency  Correction 


Figure  2  The  sharpened  images  by  pseudo  inverse,  color  normalization  and  frequency  correction  are 
shown  at  0,75  pm 
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Figure  3  A  Spatial-spectral  analysis  approach 
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Figure  4  RGB  image  of  HSI,  baekground  elassifieation  and  anomaly  detection  derived  also  from  the 
unsharpened  HSI 
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Figure  5  Edge  detection  on  HPI,  background  classification  and  target  detection  on  sharpened  HSI 
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*  Object  material 
identified 

-  Spectral  matched 
filtering 

*  Object  size,  shape 
and  orientation 
determined 

-  Spatial  filtering  of 
edges  from  HPI 


Figure  6  Fabric  and  additional  object  identification 


